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ABSTRACT 


A comic generation system controls the generation of comic 
panels. A comic panel is a graphical representation of an 
instance in time of a sequential course of events. Each 
graphical representation typically includes a graphical rep- 
resentation of at least one character. The character has 
gestures that reflect a body pose and expressions that reflect 
a facial pose. Moreover, the character may have text asso- 
ciated with it (i.e., the character "speaks"), which is dis- 
played in the comic panel within a bounded area called a 
"balloon." The comic generation system receives an inter- 
action event and generates a comic panel. Examples of 
interaction events include text input by a user and selection 
of gestures and expressions by a user. When the interaction 
event provides text, the comic generation system automati- 
cally searches the text to identify any words or phrases that 
give rise to gestures and expressions for the character 
representing the user who provided the interaction event. 
Moreover, the comic generation system searches the text for 
trigger words that indicate alterations to the comic panel. 
The comic generation system generates the comic panel 
based on the results of its searches. The comic generation 
system also determines which characters to place in a comic 
panel, the positions of the characters, and the orientations of 
the characters. The comic generation system additionally 
determines the placement of balloons containing text. Also, 
the comic generation system determines when to generate a 
new comic panel and when to modify a comic panel that has 
already been generated. Furthermore, the comic generation 
system maintains a graphical history of the comic panels. 

7 Claims, 12 Drawing Sheets 
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METHOD AND SYSTEM FOR GENERATING device, the bodies of previous balloons appearing on the 

COMIC PANELS display are scrolled upwards, and a new balloon is added. 

The body of the new balloon is displayed on the display 

RELATED APPLICATION screen vertically above the character that spoke and below 

„. .... . - . 5 the other balloon bodies. In this conventional system, when 

This application is a continuation of prior application Ser. . , . . ...... 

it no/ ;[- C(:n fil , x . 0 inft£ v {T c „ . XT a character is moved, its balloon remains in its original 
No. 08/612,560, filed Mar. 8, 1996, now U.S. Pat. No. n,. A xtU * . , - . , . 

* Jo* • u c l £ ■ , I r . . , . , . place. Therefore, when a character is moved, it may not be 
6,069,622, priority from the filing date of which is hereby ui * j . ■ u- u . _* *■ i 1 

• j a « I. o r> possible to determine which text a particular character 
claimed under 35 U.S.C. §120. , XM . ... , 4 

* spoke. Moreover, m this conventional system, users cannot 

TECHNICAL FIELD 30 scro ^ trough tne balloons. Therefore, when text is scrolled 

off of the display screen, the user cannot view it later. 
The present invention relates generally to data processing The participation of users in chat sessions may be inter- 
systems, and, more particularly, to generating comic panels. rupted by the users having to perform other tasks. To enable 

users to view a "history" of the discussion for the period the 
BACKGROUND OF THE INVENTION is user was not participating in the chat session, some corn- 
Many computer users are using their computer systems to P uter s Y stems store ^ e te *t of the conversation. A user can 
communicate with each other in chat sessions. Each user lhen view the texl 10 review the discussion that the user 
participating in a chat session transmits messages containing missed. However, because the text alone often does not 
text to all other users. The computer system of each chat indicate the personality, the emotions, or the actions of a 
session participant displays the transmitted messages from 20 user » a mstor y containing only the text may be inadequate in 
other participants on a display screen. In some conventional aidin S a ^ t0 § ain a complete picture of the chat session, 
systems, each user participating in the chat session is rep- In some conventional computer systems, the history pro- 
resented by a graphical representation (i.e., a character). Vldes a graphical representation of a course of events. In 
Examples of a character are graphical representations of a particular, one computer system provides a history of the 
woman or of a man. A user often selects a character that 25 actions performed by an application program by displaying 
provides an indication of the user's personality. In addition, graphical representations of a display screen for instances in 
each character may be capable of exhibiting gestures and lime durin S the execution of the application program. For 
expressions that reflect the emotions of the user represented example, the history provides a graphical representation of 
by the character. As used in this context, a "gesture" refers lhe display screen prior to an action performed by the 
to a character's body pose, and an "expression" refers to a 30 application program, and the history shows another graphi- 
character's facial pose. Some conventional computer sys- cal representation of the display screen just after the action 
terns enable users to select gestures and expressions for was performed by the application program. However, this 
characters history provides the output of only a single application 

Some conventional computer systems allow more than 35 P r0 * ram exeCUtin * at a sin § le Com P uter s y stem ' 
one user's character to be displayed on a display screen SUMMARY OF THE INVENTION 

simultaneously. Typically, users can place their characters on In accordance with a first aspect of the present invention, 

different portions of the display screen. It is possible that a method is practiced in a data processing system having a 

users at different computer systems place their characters on display device . data processing system is connected to 

the same portion of the display screen simultaneously. In this ^ at least one other data processing system. In accordance with 

case, the characters are displayed overlapping each other. this methodj a plurality of comic pancls are provided that 

The user typically is responsible for positioning the charac- represent a sequential course of events. Each comic panel 

ter on the display screen so that the character is not displayed prov ides a graphical representation of an instance in time of 

overlapping another character. the sequential course of events. In addition, each graphical 

The text of a message is typically displayed on the display 45 representation includes a graphical representation of at least 

device. Some conventional systems place this text in a one character. Moreover, an interaction event is received that 

window that has a scroll bar. A user can scroll through the indicates a modification of a comic panel. A comic panel 

text in this window using the scroll bar. Other conventional reflecting the interaction event is automatically generated at 

systems display the text above the characters on the display the data processing system. Then the generated comic panel 

device. Yet other conventional systems display the text in a 50 is displayed on the display device at the data processing 

"balloon." The text of a message is often displayed in a system. 

bounded area with a border, called a "balloon." For example, \ n accordance with a second aspect of the present 

a balloon may take on the form of an oval. The balloon has invention, a method is practiced in a data processing system 

a balloon body that contains the text and a balloon tail that having a display device. The data processing system is 

indicates which character is "speaking" the text. 55 connected to at least one other data processing system. In 

In one conventional computer system, when a character accordance with this method, at least one comic panel is 

speaks, the computer system displays the character on the provided. The comic panel provides a graphical representa- 

display screen. The system also adds a balloon for encap- tion of an instance in time of a sequential course of events, 

sulating the text of the character's speech. Each time that a The graphical representation includes a graphical represen- 

character speaks, a new balloon appears that encapsulates 60 tation of at least one character that represents a user of one 

this text, and this new balloon replaces the previous balloon of the data processing systems. A current comic panel is 

for that character. In another conventional system, when a displayed on the display device at each data processing 

character speaks, a balloon is displayed for that character. As system. The current comic panel represents a most recent 

additional characters speak, balloons are generated for their event relative to each other event in the sequential course of 

text. However, only the tail of the most recently generated 65 events. An interaction is received that indicates a modifica- 

balloon is displayed. That is, for each character that speaks, tion of a comic panel. After the interaction event is received, 

the tail is removed from the balloon last added to the display it is determined whether to generate a new comic panel 
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incorporating the modification indicated by the interaction device. An interaction event is received. A comic panel is 

event. Upon determining to generate a new comic panel, a automatically generated based on the received interaction 

new comic panel is generated incorporating the modification event. The comic panel provides a graphical representation 

indicated by the interaction event. However, upon determin- of an instance of time during a sequential course of events, 

ing to not generate a new comic panel, the current comic 5 The generated comic panel is displayed on the display 

panel is modified to incorporate the modification indicated device. 

by the interaction event. In either case, the comic panel In accordance with another aspect of the present 

incorporating the modification indicated by the interaction invention, a data processing system includes a display 

event is displayed at the data processing system. device. The data processing system also includes a comic 

In accordance with an additional aspect of the present 1Q generation system for generating comic panels. Each comic 

invention, a method is practiced in a data processing system panel provides a graphical representation of an instance in 

having a display device. It is determined with the data time of a sequential course of events. Each graphical rep- 

processing system that a new comic panel is to be generated. resentation includes a graphical representation of at least one 

The comic panel provides a graphical representation of an character. The comic generation system includes a receiver 

instance in time of a sequential course of events. Each for receiving an interaction event indicating a modification 

graphical representation includes a graphical representation 15 of a comic panel. The comic generation system also includes 

of at least one character. Upon determining that a new comic a generator for generating a comic panel incorporating the 

panel is to be generated, the characters that are to be interaction event received by the receiver. Moreover, the 

displayed in the new comic panel are determined. Then, a comic generation system includes a displayer for displaying 

placement for each character in the new comic panel is the comic panel generated by the generator, 

determined, where the placement indicates a sequential 20 i n yet another aspect of the present invention, a first data 

position of a character relative to each other character in the processing system connected to a second data processing 

comic panel. Next, a new orientation for each character system includes a display device. The first data processing 

relative to each other character is determined. Then the new system also includes a comic generation system for gener- 

comic panel is generated, and the generated new comic ating comic panels. Each comic panel provides a graphical 

panel is displayed on the display device at the data process- 2 5 representation of an instance in time of a sequential course 

ing system. of events. Each graphical representation includes a graphical 

In accordance with another aspect of the present representation of at least one character that represents a user 
invention, a method is practiced in a data processing system of one of the data processing systems. The comic generation 
having a display. A comic panel that provides a graphical system includes a receiver for receiving an interaction event 
representation of an instance in time of a sequential course 30 from a user indicating a modification of a comic panel. The 
of events is provided. The graphical representation includes comic generation system also includes a generator for gen- 
a graphical representation of at least one character. Textual erating a new comic panel incorporating the interaction 
input is received for a character. The received textual input event received by the receiver. Moreover, the comic gen- 
is placed within a balloon having a balloon outline. The eration system includes a displayer for displaying the new 
balloon outline includes a balloon body containing the 35 comic panel generated by the generator at the first data 
textual input and a balloon tail indicating the character to processing system. 

which the textual input corresponds. A position of the In accordance with a further aspect of the present 

balloon containing the textual input is determined in the invention, a data processing system includes a display 

comic panel, where the balloon does not interfere with the device, a server application program, and a client application 

display of any balloon already positioned in the comic panel. 40 program. An interaction event is received at the server 

Then the balloon containing the textual input is displayed in application program from the client application program. A 

the comic panel on the display device at the data processing comic panel is automatically generated with the server 

system. application program based on the received interaction event. 

In accordance with yet another aspect of the present The comic panel provides a graphical representation of an 

invention, a method is practiced in a data processing system 45 instance in time during or sequential course of events. The 

having a display device. A plurality of comic panels are graphic representation also includes at least one graphical 

provided. The plurality of comic panels represent a sequen- representation of a character. The generated comic panel is 

tial course of events. Each of the comic panels provides a forwarded from the server application program to the client 

graphical representation of an instance in time of the sequen- application program. 

tial course of events. Each graphical representation includes 50 In accordance with another aspect of the present 
a graphical representation of at least one character. A scroll invention, a data processing system includes a display 
bar displayed on the display device is provided for scrolling device, a server application program, and a client application 
through the plurality of comic panels. The scroll bar includes program. The data processing system is connected to at least 
a scroll box having a position in the scroll bar, where the one other data processing system, and each connected data 
position corresponds to one of the plurality of comic panels. 55 processing system has client application program. Under 
A current comic panel is displayed on the display device at control of the client application program of the data pro- 
each data processing system. The current comic panel rep- cessing system, an interaction event is received. Then a 
resents a most recent event relative to each other event in the comic panel is generated. The comic panel provides a 
sequential course of events. The scroll box on the scroll bar graphical representation of an instance in time of a sequen- 
has a current position corresponding to the current comic 60 tial course of events. Each graphical representation includes 
panel. An indication is received from a user to reposition the a graphical representation of at least one character. Next, the 
scroll box in the scroll bar. A comic panel different from the generated comic panel is displayed at the data processing 
previous comic panel is displayed on the display device after system. Moreover, the received interaction event is for- 
the indication to reposition the scroll box in the scroll bar is warded to the server application program. In addition, the 
received. 65 server application program forwards the received interaction 
In yet a further aspect of the present invention, a method event to each other client application program at each 
is practiced in a data processing system having a display connected data processing system. 
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In accordance with yet a further aspect of the present acters to place in a panel, the positions of the characters, and 
invention, a computer- readable storage medium, upon which the orientations of the characters. The CG system addition- 
is stored a comic generation program for automatically ally determines the placement of balloons containing speech 
generating comic panels is provided. The comic generation text. Furthermore, the CG system maintains a graphical 
program displays at least one comic panel. The comic panel 5 history of the comic panels that are generated, 
provides a graphical representation of an instance in time of FIG. 1 is a block diagram of a data processing system 
a sequential course of events. The comic generation program which is suitable for practicing the preferred embodiment of 
receives an interaction event indicating a modification of the preset invention. The data processing system 100 
one of the comic panels. Then, the comic generation pro- includes a CPU 102 that is connected to input devices 104 
gram generates a comic panel based on the received inter- 10 and 0Ut P ut devices 106. The data processing system also 
action event. Hie comic generation program then displays includes a network interface 110 connected to a network 
the generated comic panel on the display device. 112 for interfacing the computer system with the 

network. Memory 108 stores data and code. Specifically, 

BRIEF DESCRIPTION OF THE DRAWINGS memory 108 holds a copy of an operating system 116, and 

15 a comic generation system 118. A connecting mechanism 

FIG. 1 is a block diagram of a data processing system S uch as a bus, connects the components of the computer 

which is suitable for practicing the preferred embodiment of system. 

the present invention. FIG. 2 illustrates the inputs and outputs to the CG system 

FIG. 2 illustrates the inputs and outputs to the comic 118. An interaction event receiver 202 receives interaction 

generation system. 20 events from users. One skilled in the art will recognize that 

FIG. 3 illustrates a dialog box provided by the comic the interaction event receiver can receive interaction events 

generation system to enable selection of a character. from another source, such as an application program. The 

FIG. 4 illustrates a user interface provided by the comic interaction event receiver forwards the interaction event 204 

generator system, t0 tne CG system 118. The CG system generates a comic 

m • a . _ . <■ * ->s panel description 118 describing the comic panel and for- 

FTG. 5 is a flowchart providing an overview or the steps , iL . , ... . . , f m 

r j . *u • * * wards this description to a comic panel Tenderer 210. 1 tie 

performed by the comic generation system to generate , . . . . c . 

r . i j • l * • description is used to generate a bitmap of a comic panel, 

comic panels during a chat session. ™_ 1 . . , & , . , 

r ° The comic panel Tenderer 210 displays the comic panel on 

FIG. 6 is a flowchart illustrating the steps performed to a dis p lay device 0ne skm&d in the art wiu recognize tnat the 

generate a comic panel. 3Q comic panel Tenderer can display the comic panel at another 

FIG. 7 illustrates three comic panels that show how the device, such as a printer, 

comic panel changes based upon speech text. Initially, when a user invokes the CG system 118, the CG 

FIG. 8 is a flowchart providing an overview of the steps system provides the user with a dialog box for selecting a 

performed to place characters in a comic panel. character. FIG. 3 illustrates this dialog box 300. The CG 

FIG. 9 is a flowchart illustrating the steps performed to 35 system requests a user to identify a network server 302 to 

select characters to place in a comic panel. connect to. The CG system also requests the user to select a 

FIG. 10 illustrates sample balloons. character. In particular, the CG system provides a list of 

_ . , .„ , _ , names of characters (e.g., Tongue Tyed) for predefined 

HG. U is a flowchart Ulustrating the steps performed to characlers . M 

is provided with a drop-down list box 

place balloons in the comic panel. 40 m ^ a uger a name ^ the CG system 

FIG. 12 provides comic panels that illustrate routing displays a bitmap 308 of the character. The user is then able 

channels. t0 view the bitmap of the character before selecting that 

DETAILED DESCRIPTION OF THE t^^^^£^^T£lZ^ 

INVENTION 45 user , s feal Qame 312 WneQ a single user k using tne qq 

The preferred embodiment of the present invention pro- system to generate comic panels, the user can use the dialog 

vides a comic generation (CG) system for generating comic box multiple times to select multiple characters, 

panels to represent communication between multiple char- After the user has provided the requested user 

acters. A user at a single computer system can generate information, the CG system 118 displays a user interface for 

comic panels using the CG system during a comic session. 50 using the CG system. FIG. 4 illustrates this user interface 

In addition, users at multiple computer systems may com- 400. The user interface is divided into several display areas, 

municate with each other using the CG system by transmit- In one display area 410, the CG system displays generated 

ting messages to each other through computer systems comic panels from the session. This display area 410 illus- 

connected by a network during a chat session. Similarly, trates a sample series of four comic panels. Each comic 

users may communicate with each other using the CG 55 panel contains one or more characters. For example, in 

system while time-sharing. In particular, a user provides comic panel 412, there are two characters, a female character 

interaction events to the CG system. The interaction events 414 and a male character 416. When a character "speaks," 

include speech text, selection of gestures and expressions, the spoken words are placed in a balloon 418. The comic 

and the entrance and exiting of users from the chat session. panel also has a background 420 and a panel frame 422. 

Upon receiving an interaction event, the CG system gener- 60 Although FIG. 4 illustrates comic panels that are rectangular 

ates a comic panel or modifies a currently existing comic and of equal size, one skilled in the art will recognize that 

panel. When the interaction event provides text, the CG the comic panels can be different shapes and sizes than those 

system searches the text to identify any words or phrases that shown in FIG. 4. Moreover, the CG system provides various 

give rise to gestures and expressions for the character types of balloons including speech, thought and whisper. In 

speaking the text. Moreover, the CG system searches the text 65 one display area 430, the CG system displays text 432 

for trigger words that indicate alterations to the comic panel. entered by the user, along with buttons 434 that enable a user 

The CG system also automatically determines which char- to select the type of balloon in which the text is to be 
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displayed. When a user is participating in a chat session and The CG system 118 receives an interaction event (step 502). 

enters text, the CG system associates the text with the The CG system stores a description of each comic panel it 

character that represents the user. When a single user is using generates for a session (i.e., a chat session or a comic 

the CG system to generate comic panels, the user selects a session). The CG system retrieves the description of the last 

character to associate with text that is entered using a dialog 5 comic panel that it generated (step 504), which is known as 

box provided by the CG system, the current comic panel. Initially, when a user starts a session 

An additional display area 440 of the user interface and when there are no comic panels that have been 

contains a character 442 who is speaking the text entered by generated, the CG system skips step 504 and just generates 

the user. The CG system generates the character with a new comic panel. Then, the CG system generates a comic 

gestures and expressions as a user enters speech text. A user 1Q panel (step 506), which will be discussed in further detail 

can alter the character that is automatically generated by the below relative to FIG. 6. 

CG system. For example, the CG system automatically Nexl> the CG system determines whether to replace the 

selects gestures and expressions for a character based on current comic j or a(ld a new comic , tQ the dis { 

speech text and displays the character with the selected saccn ( 508) The CG m i(M m {Q 

gesture and expression at only the computer system of the ^ fl mmaX ^ x than ' * new 

user represented by the character. A user can select gestures 13 r . . IT ,. r ^ ... 

and expressions to override the ones selected by the CG COmi ? P ai *l ; However, the CG system starts a new cormc 

system through use of a user interface such as a dialog box. P anel wh u en me necessary characters (he characters neces- 

When a user does so, the CG system modifies the comic sar y 10 j he comic P anel > descnbed * &rther detail below), 

panel for the user to view. Then, the user can request that the do not fit mt0 the current °° mic P anel or when the balloons 

comic panel be generated. This display area 440 also dis- 20 for these characters do not fit into the current comic panel, 

plays a bitmap of the head portion of up to ten characters 444 Ak°> tne CG system does not display a character in a comic 

that appear most frequently in the comic panels for the P ane l with rnore than one balloon. Therefore, the CG system 

session. One skilled in the art would recognize that the use starts a new comic panel when a character speaking in the 

of ten characters is illustrative and that more or fewer current panel speaks again. Moreover, the CG system starts 

characters can be displayed. 2 5 a new panel when modifying the current comic panel would 

When users are participating in a chat session, a CG overwrite a gesture or expression of a character in the comic 

system 118 resides in a client at each user's computer panel. When the CG system is to replace the current comic 

system. In addition, a single comic server 112 is connected panel, the CG system overwrites the current comic panel 

to each user's computer system through the network. When with the generated comic panel (step 510); otherwise the CG 

a user transmits a message that contains interaction events to 30 system adds the generated comic panel to the comic panels 

the CG system, the CG system uses the message to generate that are already displayed on the display screen (step 512). 

a comic panel or modify an existing one. The CG system FIG. 6 is a flowchart illustrating the steps performed to 

also forwards the message to the comic server. In turn, the generate a comic panel. First, the CG system places a 

comic server forwards the message to each other comic background in a comic panel (step 602). The CG system has 

generation system at each other user's computer system. 35 a set of bitmaps to use as backgrounds, and the CG system 

Then, each of these comic generation systems generates a selects a particular background for the session, such as the 

comic panel or modifies an existing comic panel using the inside of a house. When a comic server is available, the 

message. One skilled in the art would recognize that use of comic server can select a background. Additionally, the CG 

a single comic server is only illustrative and several comic system modifies the comic panel based on the text received 

servers can be connected to each other, with each comic 40 from users. In particular, the CG system searches the text for 

generation system connected to one of the servers. predefined trigger words, such as the names of states, and 

Moreover, rather than having the CG system reside in the modifies the comic panel to reflect the semantics of the 

client, one skilled in the art would recognize that the CG words (i.e., their contextual meanings). The modifications 

system could reside in the comic server and the comic server can include changing the background, adding an element, or 

could generate a comic panel for forwarding to each user's 45 adding a character to the comic panel. These modifications 

computer system. When a single user at one computer are called semantic elements. The CG system can typically 

system is using the CG system to generate comic panels, the modify the comic panel in several ways for each trigger 

user provides interaction events to the CG system and the word. To provide variety in the display of the comic panels, 

CG system generates a comic panel, without forwarding the CG system cycles through the possible modifications as 

these interaction events to a comic server. 50 the same trigger word is encountered multiple times. FIG. 7 

As discussed above, the CG system 118 provides a illustrates three comic panels that show different types of 

predetermined number of characters. Each character is rep- semantic elements. In particular, in each of the three comic 

resented in memory 108 by bitmaps. Each character includes panels, the male character states that he is from Ohio, and 

a separate bitmap for its body portion and head portion. By the word "Ohio" is a predefined trigger word. In comic panel 

providing separate head portions and body portions, the CG 55 700, the CG system modifies the background 702 to show a 

system enables expressions and gestures to be independent map of Ohio. In comic panel 710, the CG system modifies 

of each other. When displaying the character, the CG system the comic panel to include a "Go Browns" banner 712. In 

provides a "white" outline around the body 454 (i.e., a body comic panel 720, the CG system modifies the comic panel 

halo) and a "white" outline around the head 452 (i.e., a head to include an additional character 722 who is asking a riddle, 

halo) in order to distinguish the character from the back- 60 The CG system next places characters in the comic panel 

ground. The CG system typically displays at most five (step 504). FIG. 8 is a flowchart providing an overview of 

characters in each comic panel. One skilled in the art will the steps performed to place characters in a comic panel, 

recognize that this limit of five characters is intended solely Each of these steps will be described in more detail below, 

to be illustrative, and will recognize that alternative embodi- A session may include comic panels having multiple char- 

ments may employ more or fewer characters per panel. 65 acters. Because the CG system displays at most five char- 

FIG. 5 is a flowchart providing an overview of the steps acters in a comic panel, the CG system selects at most five 

performed by the CG system in generating comic panels. characters to display (step 802). After selecting the 


07/08/2004, EAST Version: 1.4.1 


US 6,232,966 Bl 

9 10 

characters, the CG system selects gestures for the characters CG system generates the character with an expression 

(step 804). The CG system also selects expressions for the indicating the character is laughing. In addition, the acronym 

characters (step 806). Then the CG system selects the IMHO stands for in my humble opinion. Upon finding this 

positions of the characters in the comic panel relative to each acronym, the CG system generates the character with the 

other character (step 808). Next, the CG system selects the 5 gesture of pointing to itself. The CG system also determines 

orientations of the characters (step 810). The orientations that text that is capitalized indicates shouting, and generates 

indicate the directions which the characters face relative to the character with an expression indicating the character is 

each other. shouting. In addition, the CG system recognizes certain 

FIG. 9 is a flowchart illustrating the steps performed to predefined greetings, including "hello" and "good-bye." The 

select characters to place in a comic panel (see step 802 in 30 CG system also maintains a list of these greetings along with 

FIG. 8). The CG system selects each character in the session, associated gestures and expressions. Upon finding a 

starting with the first one in a list of characters that it greeting, such as "hello," in the text, the CG system gcner- 

main tains (step 902). Characters are classified as necessary ates the character with an associated gesture, such as one of 

or allowed. If the character is speaking (step 904), the CG waving. Moreover, the CG system searches the text for 

system identifies that character as necessary (step 906). 35 pronouns, such as to "I" and "you." When the text references 

When a character is identified as necessary, the CG system the character (i.e., includes "I"), the CG system generates 

must place the character in the comic panel. If the character the character with the gesture of pointing to itself. When the 

has a new gesture or expression (step 908), the CG system text references another character, the CG system generates 

identifies that character as necessary (step 906). If the the character with the gesture of pointing to another, 

character is newly being addressed (step 910), the CG 2 o When the CG system finds multiple indicators of gestures 

system identifies that character as necessary (step 906). If and expressions, the CG system attempts to draw each of 

the character has newly entered the chat session (step 912), them. For example, the text "I can't make it :-(." results in 

the CG system identifies that character as necessary (step a character with the gesture of pointing to itself and an 

906). When users are participating in a chat session, by expression of frowning. Also, the emoticon ":-(" results in 

displaying a character who has entered the chat session, the 2 s an expression indicating sadness with a gesture of hunched 

CG system notifies each user participating in the chat shoulders. When the CG system determines there is a 

session of the new participant. If the character was near any conflict between multiple gestures or expressions, the CG 

of the necessary characters in recent comic panels or con- system determines which ones to use based on a predefined 

tinues to be addressed from a prior comic panel (step 914), priority scheme (i.e., the priority scheme has been defined by 

the CG system identifies the character as allowable (step 30 the developers of the comic generation system). One skilled 

916), An allowable character may be displayed in the comic in the art will recognize that alternate priority schemes may 

panel, for example, when there are fewer than five necessary be used. For example, the emoticon ":-(" has a low priority 

characters, but an allowable character is not necessary. If all for a gesture indicating sadness (e.g., hunched shoulders), 

characters have been selected to be identified as necessary or while the text "good-bye" has high priority for a gesture of 

allowable (step 918), the CG system selects characters to 35 waving. Therefore, if the text contains "good-bye :-(" the 

display in the comic panel from among the identified char- CG system displays a character with a gesture of waving 

acters (step 922), otherwise the CG system selects the next instead of with a gesture of hunched shoulders, 

character to identify (step 920). Gestures and expressions often represent emotions. Each 

The CG system of the disclosed embodiment selects at emotion ranges from neutral (i.e., no emotion) to very 

most five of the identified characters to display in the comic 40 strong. For example, a character can be angry, and shown 

panel. The characters identified as necessary take priority with an expression indicating the character is shouting, or 

over characters identified as allowable. When more than five very angry, and shown with an expression indicating the 

characters are found to be necessary, the CG system gener- character is shouting along with a gesture of waving a fist, 

ates a new comic panel with only the character representing When no expressions or gestures are indicated by the text, 

the user who provided an interaction event after the last 45 the CG system selects a neutral expression and a neutral 

generated comic panel was displayed. Then, each other gesture. The CG system has multiple neutral expressions and 

character is added to the comic panel according to the above gestures for each emotion. In order to increase the variety in 

discussed steps. For example, in FIG. 4, in comic panel 412, the comic panels, the CG system cycles through its list of 

the male character is identified as necessary because that neutral expressions and gestures. Moreover, for a particular 

character is speaking and the female character is identified 50 emotion represented by a gesture or expression, different 

as necessary because she is being newly addressed, characters may have different gestures or expressions. For 

The CG system selects the gestures and expressions based example, a male character saying "hello" may be shown 

on the text entered by a user (see steps 804 and 806 in FIG. with a gesture of waving, and a female character saying 

8). The CG system searches the text for emoticons, which "hello" may be shown with a gesture of curtseying, 

are a series of characters that represent emotions. The CG 55 Once the CG system has determined which characters will 

system maintains a list of emoticons along with their asso- be included in the comic panel, the CG system determines 

dated gestures and expressions. For example, the emoticon the position of each character in the comic panel as well as 

":-)" indicates being happy and the emoticon ":-(" indicates the orientation of each character (see steps 808 and 810 in 

being sad. Upon finding these emoticons, the CG system FIG. 8). In particular, the CG system positions characters 

modifies the expression of the character to indicate happi- 60 who are speaking to each other near each other in a comic 

ness or sadness, respectively. Moreover, the CG system panel. The CG system also attempts to position characters so 

maintains a list of predefined acronyms along with associ- that the fewest number of characters are moved to a new 

ated gestures and expressions. The CG system searches the position relative to their position in the last generated comic 

text for these acronyms, and upon finding these, the CG panel. Also, the CG system determines which characters are 

system displays the character with the associated gestures 65 speaking to each other by searching the text for names of 

and expressions. For example, the acronym LOL stands for characters. The user can select a character to address by 

laughing out loud. Upon finding this acronym in the text, the selecting the character with an input device 104, such as a 
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mouse. If it is not clear that the text is addressed to a 
particular character, the CG system interprets the text as a 
comment to all of the other characters. The CG system 
positions the characters by positioning each character, then 
positioning each other character without moving any char- 5 
acter that has already been positioned. Furthermore, the CG 
system determines the orientation of each character based on 
which characters are speaking to each other. The CG system 
maintains a list of characters and to whom that character is 
speaking to determine the position and the orientation of 1Q 
each character. The CG system flips bits in the bitmap of the 
character to provide a different orientation. For example, 
comic panel 470 of FIG. 4 shows that the characters speak- 
ing to each other are positioned by each other and are 
oriented relative to each other ("face each other"). J5 

The CG system also places balloons in the comic panel 
(step 606). The CG system displays the text "spoken" by the 
characters in balloons. FIG. 10 illustrates sample balloons. 
There are different types of balloons, including speech 
balloons, thought balloons, and whisper balloons. One 20 
skilled in the art will recognize that other types of balloons, 
such as shout balloons, can be used. Each balloon has a 
balloon body that contains the text and a balloon tail that is 
used to identify the character speaking the text. Each of the 
types of balloons have different balloon outlines that identify 2 $ 
the type of the balloon. A speech balloon 1010 contains 
typical speech. The balloon outline of the speech balloon is 
solid, with the balloon body containing the text, while the 
balloon tail is attached to the balloon body. A thought 
balloon 1020 indicates that a character is thinking. The 30 
thought balloon has a solid balloon outline, with the balloon 
body containing text, and the balloon tail consisting of a 
series of increasingly larger ovals. A whisper balloon 1030 
indicates a character is whispering. The whisper balloon has 
a dashed outline for the balloon body and the balloon tail, 35 
and its text is italicized. A whisper balloon indicates that the 
user wishes to transmit the text to only particular other users. 
The CG system enables a user to identify the users who are 
to receive the text. 

FIG. 11 is a flowchart illustrating the steps performed to 40 
place balloons in the comic panel. First, the CG system 
places the balloon bodies and routing channels (step 1102). 
The routing channel is the width of space in which the 
balloon tail can be placed without overlapping any other 
balloon tail or balloon body. Next, the CG system places the 45 
balloon tails in the comic panel (step 1106). These steps will 
now be described in more detail below. 

The CG system places a balloon body so that part of it is 
centered over a character. Also, all balloon bodies in a comic 
panel are placed above the tallest character. Moreover, 50 
because the relative placement of balloons determines the 
reading order, the CG system places the balloons so that the 
user viewing the comic panel can read the balloons in the 
order in which the characters spoke. In particular, balloons 
are read in a top-down order, and when multiple balloons are 55 
at the same height, they are read from left to right (i.e., they 
are read in a spoken order). The order in which balloons are 
read is independent of where characters are placed in the 
panel. 

Once a balloon body has been placed, it is not moved. 60 
Once each character's balloon body has been placed, the CG 
system places the balloon tails. Each balloon tail is placed in 
a routing channel, which is an area in which a particular 
balloon tail is placed. A routing channel cannot overlap any 
other routing channel. FIG. 12 provides comic panels that 65 
illustrate routing channels. For example, in FIG. 12, comic 
panel 1200 illustrates one routing channels 1202. Initially, a 
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routing channel 1202 is allocated for one character's balloon 
body 1204. Then, when placing the next character's balloon 
body 1206, the CG system determines a routing channel 
1224 that decreases the size of the routing channel 1202 to 
be the size of routing channel 1222 for balloon body 1204. 
When a routing channel of one character's balloon body is 
encompassed by another character's balloon body, such as 
with routing channel 1222 and balloon body 1226, the CG 
system repositions the balloon body so that it does not 
encompass the routing channel. Then, the CG system rede- 
termines the routing channels and places the balloon tails in 
the comic panel. For example, in comic panel 1230, balloon 
body 1226 has been repositioned, routing channel 1222 has 
decreased in size to become routing channel 1234, and 
routing channel 1236 has been added for balloon body 1226. 

Sample pseudo code is provided below to show the steps 
performed to determine the placement of the balloon bodies. 
In the pseudo code, the variable B=(B 1 , . . . , BJ represents 
an array of balloons ordered according to the order in which 
the text the balloons contain were spoken. In addition, the 
variable R=(Ri . . . RJ represents a corresponding array of 
routing channels, with the width of each routing channel R t 
defined by left and right endpoints R,.l and R,.r. The display 
screen typically is represented in (X,Y) coordinates, with the 
top, left corner of the display screen having the coordinates 
(0,0). The X-coordinate of the center of a character's head 
is represented by x r The following PlaceBalloons function 
computes the horizontal placement for an array of n 
balloons, and sets up the routing channels R a . . . R„. It 
returns the number of balloons that are placed successfully. 


Function Place Bal loons (B,R,x,T) 
for j » 1 to n do 

Wj- FindWidth(Bj); 
Rj := [Xj-Wj-, xj+wjf 
for i = 1 to j-1 do: 

Rj MaxAllowable (K u x u Rj, x,-); 

end for 

if width (Rj) >- Wj then 

Rj :- Position (B,-, Rj) 

else 

if SSqueezeBalloon (Rj, Tj) then 
return j-1; 

end if 

end if 

for i»l to j-1 do 

R r :» ReduceChannet (Ri, Xj, Rj, Xj) 

end for 

end for 
return n 
end Function 


The PlaceBalloons function selects each balloon, placing 
each selected balloon in a comic panel. For example, after 
placing several balloons, to place a new balloon, the Place- 
Balloons function selects a target width for the new balloon 
by invoking the Find Width function. The target width is at 
least as wide as the widest word. Also, the target width 
ranges from the width of the balloon body when text is 
placed in the balloon body in as few lines as possible, given 
the width of the comic panel, to the width of the balloon 
body when text is placed in the balloon body in as many 
lines as possible, given vertical space that remains for new 
balloon bodies in the comic panel. Then, for a particular 
balloon body, the PlaceBalloons function selects a routing 
channel that encompasses the width of the balloon body. The 
PlaceBalloons function invokes the MaxAllowable function 
which examines the width and position of each previously 
placed routing channel to determine an allowable width and 
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position of this new balloon's routing channel. If the width 
of the available routing channel is greater than the width of 
the balloon body, then the PlaceBalloons function invokes 
the Position function to shift the balloon body over to fit 
within the routing channel, while ensuring that the balloon 5 
body remains at least partially above the head portion of its 
character. If the width of the routing channel is smaller than 
the width of the balloon body, then the balloon body is 
reduced by the SqueezeBalloon function, which is invoked 
by the PlaceBalloons function, to the maximum allowable 10 
width of its routing channel and placed in this routing 
channel. Next, for each balloon previously placed, the 
PlaceBalloon function invokes the ReduceChannel function 
to decrease the width of that balloon's routing channel by 
removing any part of the routing channel that intersects the 15 
new balloon body's routing channel. The width and hori- 
zontal position of the new balloon body is s et to be t he 
width and horizontal position of the new balloon's routing 
channel. 

After using the above pseudo code to determine the 20 
horizontal position of balloons, the CG system determines 
the vertical placement of balloons. In particular, each bal- 
loon can be placed vertically so that is no higher than the 
bottom of any balloons already placed to its right, and no 
higher than the top of any balloons already placed to its left. 25 

After placing balloon bodies based on the vertical and 
horizontal placement determinations, the CG system posi- 
tions the balloon tails in the routing channels. The CG 
system places the balloon tails so that the bottom of each tail 
is at the same height in the comic panel as the bottom of each 30 
other tail. Also, all of the balloon tails have bottoms lower 
than the bottom of the lowest placed balloon body. Also, the 
CG system uses routing channels to ensure that balloon tails 
do not overlap other balloon tails or balloon bodies. The CG 
system positions the balloon tails to identify the character 35 
who is speaking the text within the balloon. 

Next, the CG system selects a camera zoom (step 608). A 
camera zoom indicates how distant the characters in the 
comic panel appear to a user viewing the comic panel. For 
example, the characters in comic panel 460 appear more 40 
distant than the characters in comic panel 470 of FIG. 4. The 
CG system selects a camera zoom that shows each of the 
displayed characters. Also, the CG system does not select a 
camera zoom that would display a character cut at the neck 
or ankles. The camera zoom does not affect balloons. 45 
Additionally, the CG system selects a comic panel frame 
(step 610). The comic panel frame is a border around the 
comic panel, such as panel frame 422 of FIG. 4, 

Because a user participating in a session may leave the 
session and return at a later time, the CG system provides a 50 
graphical history of each session. That is, the CG system 
stores the comic panels that are generated for a session. 
Then, the CG system enables a user to display the series of 
comic panels, ordered sequentially by time, on a display 
screen. The CG system enables the user to scroll through the 55 
displayed comic panels. For example, the user interface 
illustrated in FIG. 4 includes a scroll bar 480 for scrolling 
through the comic panels in display area 410. That is, the CG 
system provides a graphical history of a session. Moreover, 
a user about to participate in a session may also find it 60 
helpful to review the graphical history to determine whether 
to participate in a particular session. 

Although the present invention has been described in 
terms of the preferred embodiment, it is not intended that the 
invention be limited to this embodiment. Modification 65 
within the spirit of the invention will be apparent to those 
skilled in the art. For example, the CG system could provide 


transitions between backgrounds, such as from inside a 
house to a swimming pool in a yard. The CG system could 
then transition from one background to another. Also, the 
CG system could display characters performing various 
activities, such as drinking tea in a house or swimming in the 
pool. Moreover, a comic panel can be generated without 
including a character. The scope of the present invention is 
defined by the claims which follow. 
What is claimed is: 

1. A data processing system, comprising: 

(a) a display device; and 

(b) a comic generator coupled to the display device for 
generating: 

(1) a comic panel that includes a graphical representa- 
tion of at least one character; and 

(2) text that corresponds to the input received for the at 
least one character; 

(3) a balloon having a tail for indicating the character 
that is associated with the input and placing the 
generated text in the balloon; and 

(4) positioning the tail between a position of the 
balloon and another position for the character in the 
comic panel, the balloons, tail and character being 
disposed at positions that are non-overlapping of any 
other positions for balloons, tails and characters that 
were previously positioned for display in the comic 
panel. 

2. The system of claim 1, further comprising a plurality of 
comic panels displayed on the display device and wherein 
the displayer displays the generated comic panel along with 
the plurality of comic panels. 

3. The system of claim 1, further comprising a scroll bar 
displayed on the display device for enabling a user to scroll 
through the plurality of comic panels, the scroll bar includ- 
ing a scroll box, the scroll box having a position in the scroll 
bar, the position corresponding to one of the plurality of 
comic panels, and wherein the displayer displays a plurality 
of comic panels based on the position of the scroll box in the 
scroll bar. 

4. The system of claim 1 wherein the generator for 
generating a comic panel modifies a last generated comic 
panel to incorporate an interaction event received by the 
receiver. 

5. The system of claim 1, further comprising when 
non-overlapping positions for displaying the balloon and tail 
in a currently displayed comic panel are unavailable, auto- 
matically generating and graphically displaying a new comic 
panel that includes the balloon, tail and the character, 
wherein the balloon, tail and the character are disposed at 
separate positions that are non-overlapping in the graphical 
display of the new comic panel. 

6. A method for automatically displaying text representing 
an input associated with a character that is graphically 
displayed in a comic panel, comprising: 

(a) generating at least one comic panel that includes at 
least one character; and 

(b) when an input associated with the character is 
received, performing actions, including: 

(i) automatically generating text that corresponds to the 
input received for the character; 

(ii) automatically generating a balloon having a tail for 
indicating the character that is associated with the 
input and automatically placing the generated text in 
the balloon; 

(iii) automatically positioning the tail between a posi- 
tion of the balloon and another position for the 
character in the comic panel, the balloon, tail and 
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character being automatically disposed at positions 
that are non-overlapping of any other positions for 
balloons, tails and characters that were previously 
positioned for display in the comic panel. 
7. The method of claim 6, further comprising when 
non-overlapping positions for displaying the balloon and tail 
in a currently displayed comic panel are unavailable, auto- 
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matically generating and graphically displaying a new comic 
panel that includes the balloon, tail and the character, 
wherein the balloon, tail and the character are disposed at 
separate positions that are non-overlapping in the graphical 
display of the new comic panel. 
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